An electronic imaging system depends on a lens system to form an image on an electronic image sensor to create an electronic representation of a visual image. Examples of such electronic image sensors include charge coupled device (CCD) image sensors and active pixel sensor (APS) devices (APS devices are often referred to as CMOS sensors because of the ability to fabricate them in a Complementary Metal Oxide Semiconductor process). A sensor comprises a two-dimensional array of individual picture element sensors, or pixels. Each pixel is typically provided with either a red, green, or blue filter, as described by Bayer in commonly assigned U.S. Pat. No. 3,971,065 issued Jul. 20, 1976, so that a color image can be produced. Regardless of electronic technology employed, e.g., CCD or CMOS, the pixel acts as a bucket in which photoelectrons are accumulated in direct proportion to amount of light that strikes the pixel during the capture of an image by the electronic imaging system.
Not all of the light that enters the front optical element of an electronic imaging system strikes a pixel. Much of the light is lost when passing through the optical path of the electronic imaging system. Typically, about 5% of the light is lost due to lens reflections and haze and about 60% is lost because of the color filter array. Moreover, some of the light strikes areas of the pixel that are not light sensitive. To gather the amount of light that is needed to make a correct exposure, the electronic imaging sensor gathers light for an interval of time called the exposure time. Based on brightness measurements of the scene to be imaged, an automatic exposure control is typically employed to determine a suitable exposure time that will yield an image with effective brightness. The dimmer the scene, the larger the amount of time the electronic imaging system needs to gather light to make a correct exposure. It is well known, however, that longer exposures can result in blurry images. This blur can be the result of objects moving in a scene. It can also be produced when the image capture device is moving relative to the scene during capture.
One method to reduce blur is to shorten the exposure time. However, this method under-exposes the electronic image sensor during image capture so dark images are generated. An analog or digital gain can be applied to the image signal to brighten the dark images, but those skilled in the art will recognize that this will result in noisy images.
Another method to reduce blur is to shorten the exposure time and preserve more of the light that passes through the optical path and direct it to the pixels of the electronic image sensor. This method can produce images with reduced blur and acceptable noise levels. However, the current industry trend in electronic imaging systems is to make imaging systems smaller and less expensive. High-grade optical elements with large apertures, which can gather more light and preserve more light passing through them, are therefore not practicable.
Another method to reduce blur is to shorten the exposure time and supplement the available light with a photographic flash. A photographic flash produces a strong light flux that is sustained for a fraction of a second and the exposure time is set to encompass the flash time. The exposure time can be set to a significantly shorter interval than without a flash since the photographic flash is strong. Therefore, the blur during the exposure is reduced. However, flash photography is only practical if the distance between the flash and the object is relatively small. Additionally, a flash adds extra cost and weight to an image capture device.
U.S. Pat. No. 6,441,848 issued Aug. 27, 2002 to Tull describes a digital camera with an electronic image sensor that removes object motion blur by monitoring the rate at which electrons are collected by each pixel. If the rate at which light strikes a pixel varies, then the brightness of the image that the pixel is viewing is assumed to be changing. When a circuit built into the sensor array detects that the image brightness is changing, the amount of charge collected is preserved and the time at which brightness change was detected is recorded. Each pixel value where exposure was stopped is adjusted to the proper value by linearly extrapolating the pixel value so that the pixel value corresponds to the dynamic range of the entire image. A disadvantage of this approach is that the extrapolated pixel values of an object that is already in motion when the exposure begins are highly uncertain. The image brightness, as seen by the sensor, never has a constant value and, therefore, the uncertainty in the extrapolated pixel values results in an image with motion artifacts. Another disadvantage is that it uses specialized hardware so that it cannot be used with the conventional electronic image sensors that are used in current commercial cameras.
Another method to reduce blur is to capture two images, one with a short exposure time, and one with a long exposure time. The short exposure time is selected so as to generate an image that is noisy, but relatively free of notion blur. The long exposure time is selected so as to generate an image that has little noise, but that can have significant motion blur. Image processing algorithms are used to combine the two captures into one final output image. Such approaches are described in U.S. Pat. No. 7,239,342, U.S. Patent Application Publication No. 2006/0017837, U.S. Patent Application Publication 2006/0187308 and U.S. Patent Application Publication 2007/0223831. The drawbacks of these approaches include a requirement for additional buffer memory to store multiple images, additional complexity to process multiple images, and difficulty resolving object motion blur.
Another method to reduce blur is to shorten exposure time and preserve more light passing through the color filter array. For silicon-based image sensors, the pixel components themselves are broadly sensitive to visible light, permitting unfiltered pixels to be suitable for capturing a monochrome image. For capturing color images, a two-dimensional pattern of filters is typically fabricated on the pattern of pixels, with different filter materials used to make individual pixels sensitive to only a portion of the visible light spectrum. An example of such a pattern of filters is the well-known Bayer color filter array pattern, as described in U.S. Pat. No. 3,971,065. The Bayer color filter array has advantages for obtaining full color images under typical conditions; however, this solution has been found to have its drawbacks. Although filters are needed to provide narrow-band spectral response, any filtering of the incident light tends to reduce the amount of light that reaches each pixel, thereby reducing the effective light sensitivity of each pixel and reducing pixel response speed.
As solutions for improving image capture under varying light conditions and for improving overall sensitivity of the imaging sensor, modifications to the familiar Bayer pattern have been disclosed. For example, commonly assigned U.S. Patent Application Publication No. 2007/0046807 entitled “Capturing Images Under Varying Lighting Conditions” by Hamilton et al. and U.S. Patent Application Publication No. 2007/0024931 entitled “Image Sensor with Improved Light Sensitivity” by Compton et al. both describe alternative sensor arrangements that combine color filters with panchromatic filter elements, spatially interleaved in some manner. With this type of solution, some portion of the image sensor detects color; the other panchromatic portion is optimized to detect light spanning the visible band for improved dynamic range and sensitivity. These solutions thus provide a pattern of pixels, some pixels with color filters (providing a narrow-band spectral response) and some without (unfiltered “panchromatic” pixels or pixels filtered to provide a broad-band spectral response). This solution is not sufficient, however, to allow high quality images without motion blur to be captured under low-light conditions because color pixels are still subject to motion blur.
Another method to reduce blur and capture images in low-light scenarios, known in the fields of astrophotography and remote sensing, is to capture two images: a panchromatic image with high spatial resolution and a multi-spectral image with low spatial resolution. The images are fused to generate a multi-spectral image with high spatial resolution. Such approaches are described in U.S. Pat. No. 7,340,099, U.S. Pat. No. 6,011,875 and U.S. Pat. No. 6,097,835. The drawbacks of these approaches include a requirement for additional buffer memory to store multiple images, and difficulty resolving object motion blur.
Another method that can be used to reduce motion blur is to use an image stabilization system having moveable lens system or special imager positioning hardware. Such systems are designed to keep the image in a stable position on the sensor. However, these systems have the disadvantage that they are complex and costly. Additionally, they do not address the case where objects in the scene are moving at different velocities.
Thus, there exists a need for producing an improved color filter array image or color image having color and panchromatic pixels, having reduced motion blur, by using conventional electronic image sensors, without the use of a photographic flash, without increasing image noise, and without significant additional cost or complexity or memory requirements.